I keep hearing that AI will replace UX researchers. I wanted to test one small part of that claim: could four AIs accurately analyze my survey data?
I surveyed 34 UX professionals. Then I analyzed the data myself, including the quantitative results, so I had findings to check the AIs' answers against. In the survey, 94% of respondents said they had found an error or unsupported conclusion in AI-generated UX research work.
I gave the same data and prompt to ChatGPT, Claude, Gemini, and Copilot. I compared their results with my verified survey analysis.
At first, the results looked promising. All four tools correctly counted respondents' roles, years of experience, frequency of AI use, and two other survey questions. But accuracy on those questions did not carry through the rest of the survey.
All four made errors counting AI tool selections. Respondents could select more than one tool. Each AI got three tool counts wrong, and all four miscounted Gemini and Perplexity.
All four made errors counting trust ratings. The survey asked about nine separate UX research activities. Copilot counted 6 of 9 correctly, Claude counted 5, and ChatGPT and Gemini each counted 4. Copilot did best on this question, but its counts for a different question-how much human verification AI work requires-were all incorrect.
The written answers posed another challenge. For the question about which UX research activity respondents trusted AI with most, I identified 94 mentions of distinct ideas. None of the four tools produced categories and counts I could use. Claude counted respondents rather than mentions. Gemini did not group the comments. Copilot grouped some themes but did not count them. ChatGPT's counts were difficult to follow and included incorrect respondent totals.
I also marked the question about least-trusted activities as a failure because the AI outputs did not provide usable groupings. I did not complete a separate detailed review of each tool's answer to that question.
The AIs seemed to struggle when they had more text and more separate items to keep track of. They handled simpler questions better, but made errors with the nine trust ratings and could not produce usable counts of the written responses.
This was one survey analyzed with one prompt, so the results do not establish how these tools perform on every dataset. They do show a practical problem: getting several questions right did not make the full analysis reliable.
I was disappointed. I hoped the AIs would help me go further with the analysis, but I spent my time checking counts and rebuilding categories. For this survey, I could not use their results without doing that work myself.
Email: theresaw@columbus.rr.com
LinkedIn: theresa-wilkinson